Automatically Extracting Subsequent Response Pages from Web Search Sources

Usually, when Web search sources such as search engines and deep Websites retrieve too many result records for a given query, they split them among several pages with, say, ten or twenty records on each page and return only the page that has the top ranked records. This page usually provides one or more hyperlinks or buttons pointing to one or more of the remaining response pages (called subsequent response pages), which inturn contain similar hyperlinks or buttons to allow users to navigate from one page to another. Information integration systems often need to access these subsequent response pages to extract the records contained in them. However, hyperlinks or buttons pointing to subsequent response pages are often displayed in different formats by different Web search sources. Due to this it becomes a challenging task to automatically identify these hyperlinks or buttons and extract the response pages referenced by them. In this paper, we propose a novel solution to automatically fetch any specified response page from autonomous and heterogeneous Web search sources for any given query. Our approach first identifies certain important hyperlinks present in the response page sampled from an input Web search source and then further analyzes them using four heuristics. Finally a wrapper is built to automatically extract any specified response page from the input source.